42390.P4424 

UNITED STATES PATENT APPLICATION 



FOR 



Image Selection based on image Content 



inventor: 

randy r. dunton 
lawrence a. booth, jr. 

prepared by: 

blakely, sokoloff, taylor & zafman 
12400 wilshire boulevard 

Seventh Floor 
los angeles, ca 90025-1026 

(408) 720-8598 



"Express Mail" mailing label numbe r £M 335^7 US 



"Express Mail Post Office to Addressee" service under 37 CFR 1.10 on the date indicated above and 
is addressed to the Assistant Commissioner of Patents and Trademarks, Washington, D.C. 20231. 



I hereby certify that this paper or fee is 



Date of Deposit } 2- 




tosited with the United States Postal Service 



(Typed or printed 




IMAGE SELECTION BASED ON IMAGE CONTENT 

« 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

5 The described invention relates to the field of imaging. In particular, the 

invention relates to the selection of an image based on the content of the image. 

2. Description of Related Art 

Image capture devices, such as still and video cameras, typically have stored 
1 0 images based upon the input of a trigger from the user. Sometimes the trigger is 
manually input, for example, by pushing a button. The trigger may alternatively be 
triggered remotely via a coupling to a flash strobe, or through a timer. 

Some events, however, are very difficult to trigger upon. As an example, 
capturing an image of lightning during a lightning storm is very difficult to capture 
15 because of the unpredictability of the event, as well as the quick timing that is 
needed to capture the event. 

One way in which lightning could be captured is by taking a video picture of 
the sky, and hoping that lightning strikes during the time the video is capturing 
images (the capture period). However, this method fails in several respects. 
20 First, the quality of video is typically not as good as that of a still picture. 

Second, because the video image must be stored, the capture period is limited to the 
amount of storage space in the camera. This is restrictive when the target image 
may take a very long time to happen. Finally, a person has to either be present 
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during the capture period to observe whether lightning had struck to know whether 
the target image was captured, or the user potentially has to review the entire video 
to see whether the desired event occurred. A more automated method of detecting a 
target image is desirable. 



SUMMARY OF THE INVENTION 

An image capture system comprises an image input and processing unit. The 
image input obtains image information which is then passed to the processing unit. 
The processing unit determines an image metric from the image information. The 
5 processing unit initiates a capture sequence when the image metric meets a 

predetermined condition. The capture sequence may store one or more images, or it 
may indicate that one or more images have been detected. In one embodiment, the 
image input comes from an image sensor, such as a CMOS or CCD sensor. In 
another embodiment, the image input comes from a video stream. 

10 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows an image capture system in accordance with one embodiment 
of the invention. 

Figure 2 shows another embodiment of an image capture system. 
5 Figure 3 shows a representation of an image sampled by the image capture 

device 201. 

Figure 4 shows a representative histogram of the luminosity corresponding to 
the image of Figure 3. 

Figure 5 shows a representation of an image that was created via a computer 
10 program. 

Figure 6 shows a representative histogram of the luminosity corresponding to 
the image of Figure 5. 

Figure 7 shows an embodiment of a triggering mechanism. 

Figure 8 shows an example of an interface that may be used on the image 
1 5 capture device. 

Figure 9 shows a graphical interface in which various thresholds are 
identified at several levels of an image metric. 

Figure 10 shows an example of a trigger system set up to measure the blue 
component of an image. 

20 
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DETAILED DESCRIPTION 

An apparatus and method for determining when one or more images match 
an image profile are described. In a first embodiment, the images are sampled by an 
image sensor in a camera. The image profile comprises one or more thresholds of 
5 image metrics such as color content and light intensity information. The images 
from the image sensor are processed and compared against the image metrics of the 
image profile. If a match is found, as will be described later, then the camera 
triggers a capture sequence in which one or more images are saved to a storage 
medium. The capture sequence may save a single image, or if desired, the capture 

1 0 sequence may save multiple images. If images are buffered, images sampled prior 
to the trigger of the capture sequence can also be saved to the storage medium, as 
will be described in more detail later. 

In a second embodiment, the images are provided from a collection of 
images or a video stream, for example, a television signal, VCR output, or digital 

1 5 video diskette (DVD) output. The images from the video stream are processed and 
compared against the image metric thresholds set up in the image profile to find a 
match. This is useful for searching a video stream for a particular image. For 
example, after a video camera is used to capture images, this second embodiment 
can be used to automatically determine whether a desired image is within the capture 

20 period. This frees the user from having to be present during the capture period. 

When an image matches the image profile, i.e., all thresholds are reached, a 
capture sequence is initiated. This may result in one or more images being stored to 
a storage medium, or it may result in an indication that the image was found in the 
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video stream. Information about the location or a time index of the identified image 
may also be saved. 

Image metrics are derived from various components of an image. In one 
embodiment image metrics include, but are not limited to, color content and light 
5 intensity information. Examples of triggering a capture sequence based upon color 
content follow. A camera focused on a tree can detect when a bluebird enters the 
image by triggering on the blue content in the image. When the blue content 
exceeds a specified threshold set up in the image profile, an image capture is 
triggered. Similarly, the camera can be set up to trigger on the bluebird leaving the 

1 0 image by monitoring the blue content in the image and triggering a capture sequence 
when the blue content falls below a specified threshold. The blue content can be 
further specified to trigger only upon a particular shade of blue in order to capture 
images of only a particular bluebird, or a particular species of bluebird. 

In a preferred embodiment, the thresholds of the image metric(s) that 

1 5 comprise the image profile can be programmed via an interface. The image profile 
can be set to readily match one or more color metrics of an image, or the image 
profile may be set so that there is rarely a match with the one or more color metrics 
of an image. For example, the trigger may be set up so that minor variations in 
image content will not trigger a capture sequence. 

20 The image profile can also be set up based upon a rate of change of an image 

metric. For example, lightning is characterized by a rapid change in light intensity, 
or luminosity. An image of lightning may be captured by specifying a rate of 
change of luminosity in the image profile. When the detected rate of change of 
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luminosity exceeds the specified rate of change of the image profile, a capture 
sequence is initiated. However, a slow rate of change of luminosity corresponding, 
for example, to a sunrise or a sunset can be set up to not trigger a capture sequence. 
Figure 1 shows an image capture system 100 in accordance with one 
5 embodiment of the invention. The image capture system includes an image input 
102, an image processing unit 104, a processor 120, and a storage medium 140. 

The image input 102 captures image information using technology that is 
well known in the art. In one embodiment, the image input 102 is a complementary 
metal oxide semiconductor (CMOS) sensor. However, the image input 102 could 

10 alternatively be a charge couple device (CCD) sensor or other type of image sensor. 
The image input may also be an interface which receives a video stream from a 
television, satellite broadcast, VCR, or other digital video recording. The image 
input 102 is coupled to provide image information to the image processing unit 104. 
Image processing unit 104 may include color look up tables for adjusting color 

1 5 values to pixel data. The image processing unit 104 may also include a compression 
unit for compressing the image information. In one embodiment, a data flow 
manager 108 couples the image processing unit 104 to a bus 1 10. 

In one embodiment, processor 120, histogram unit 130, and storage medium 
140 are coupled to bus 110. The processor 120 is coupled to receive instructions 

20 from firmware (or alternatively memory) 122 for instructing the processor what 

operations to perform. The processor optionally is also coupled to timer 124 which 
provides a timing input so that processor 120 can determine a rate of change of the 
image metric(s) generated by the histogram unit 130, as will be explained next. In 
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one embodiment, memory unit 126 is coupled to processor 120 for storing the image 
profile which includes one or more threshold values of image metric(s). The 
threshold values are compared against the image metric(s) generated by the 
histogram unit 130. 

5 Histogram unit 130 is coupled to receive image information from the image 

processing unit 104. In one embodiment, the data flow manager 108 transfers the 
image information to the histogram unit 130, and the histogram unit 130 determines 
one or more image metrics from the image information. The data flow manager may 
be a DMA, as one example. In one embodiment, the histogram unit generates 

10 colorimetric data such as the amount of red, green, and blue (RGB) components. 
Alternatively, the histogram unit 130 can determine other colorimetric data. For 
example, it can determine cyan, magenta, yellow, and light intensity (CMYK) data. 
The histogram unit 130 may also determine photometric data. Photometric data 
includes light intensity, or luminosity, measured in either absolute or relative scales. 

1 5 However, the image metrics are not limited to color and light intensity; other image 
metrics can be determined from the image information. In one embodiment, the 
image information is temporarily stored in memory 132 while the histogram unit 
130 determines image metrics. 

The processor 120 compares the image metrics generated by the histogram 

20 unit 130 with its image profile. In one embodiment, for each image metric, the 

image profile indicates a threshold value and whether a match for that image metric 
is met according to whether the measured image metric is below or above the 
threshold. Alternatively, for each image metric, two thresholds may be specified, 
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with a match occurring only if the measured image metric is above a first threshold 
and below a second threshold. Threshold levels will be discussed in more detail 
later. 

Storage medium 140 is used to store the image information when the image 
5 metrics determined from the image information matches the image profile, i.e., all 
thresholds are met. In one embodiment, one or more images are temporarily stored 
in memory area 132 prior to being stored to storage medium 140. By buffering the 
images into memory area 132, the images prior to finding a capture sequence can be 
saved. In one embodiment, various combinations of the firmware 122 and memories 
10 122, 126, 132, and 140 may be combined together within a single unit. 

External interface 160 is can be used for downloading the saved images from 
storage medium 140. External interface 160 can also be used to input the image 
profile. 

Figure 2 shows an alternate embodiment of an image capture system 200. 

15 Image capture portion 201 includes an image input 202, processor 220, firmware 
222, and timer 224 that are similar to their counterparts of the same name in Figure 
1. Image metering unit 204 incorporates image processing unit 104, data flow 
manager 108, and histogram unit 130. A host personal computer is coupled to the 
image metering unit 204 via a communication link 265. 

20 The host personal computer may provide the input of the image profile to the 

image capture portion 201 via the communication link 265. In one embodiment, a 
storage medium can be included in host personal computer 270 so that it need not 
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reside in the image capture portion 201. The host personal computer 270 may also 
implement the histogram unit 130 function in software or hardware. 

Triggering a capture sequence 
5 Figure 3 shows a representation of an image of a night-time scene sampled 

by the image capture device 201 . Figure 4 shows a representative histogram of the 
luminosity corresponding to the image of Figure 3. On the horizontal axis is the 
level of the luminosity. On the vertical axis is the count of the luminosity per each 
level. 

10 In one embodiment, a background image such as Figure 3 is captured by the 

image capture device 201. A histogram of the image determines image metric(s) 
which are then compared to subsequent sampled images. An image profile is 
generated having one or more thresholds associated with the image metric(s) of the 
background image. 

1 5 Figure 7 shows an embodiment of a triggering mechanism. At block 700, a 

histogram of image data is determined. At block 705, the count of one or more 
levels of the luminosity is compared against an image profile derived from the 
histogram of the background image. At decision block 710, if the counts of the one 
or more levels of the luminosity are greater than the image profile thresholds, then a 

20 capture sequence is initiated (block 715). However, if the comparisons are not all 
greater than the typical background image, then another image is sampled (block 
720). 
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The image profile thresholds can be preset to any amount or percentage 
above the typical background image. For example, it can be set to 5%, or 10% 
above the typical background image. Alternatively, it could be set at a preset 
amount or percentage less than typical background image. 
5 Figure 8 shows an example of an interface that may be used with the image 

capture device. A "SBI" button 802 for sampling the background image generates a 
histogram and image metrics for a background image. Preset variations based on the 
background image can be easily chosen by selecting a few buttons: one set of 
buttons 804 indicating a threshold "greater than" or "less than" the histogram of the 

10 background image, one set of buttons 806 indicating by how much (5%, 10%, or 
20%), and one set of buttons 808 indicating which image metric(s) to compare. The 
image metrics, for example, may include red, green, blue, and luminosity. 

The interface for inputting complex threshold data via buttons, however, may 
be restrictive. Thus, using a graphical interface may be more suited to the task. 

1 5 Figure 9 shows a graphical interface in which various thresholds are 

identified for several levels of an image metric. For the thresholds labeled as l >\ a 
capture sequence is triggered when an image metric at that level has a measured 
count above the threshold. For the thresholds labeled as 4 <\ a capture sequence is 
triggered when an image metric at that level has a measured count below the 

20 threshold. 

As an example, Figure 9 shows a first image metric 900 that is set up to 
trigger a capture sequence when the measured count for that image metric is above 
the threshold 902. A second image metric 910 is set up to trigger a capture sequence 
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when the measured count for that image metric is below the threshold 912. A third 
image metric 920 is set up to trigger a capture sequence when the measured count 
for that image metric is below the threshold 924 but above the threshold 922, and a 
fourth image metric 930 is set up to trigger a capture sequence when the measured 
5 count for that image metric is above the threshold 934 or below the threshold 932. 
In one embodiment, all of the image metric (e.g., 900, 910, 920 and 930) threshold 
requirements must be met to trigger the capture sequence. In another embodiment, 
only one or any combination of image metric (e.g., 900, 910, 920, or 930) threshold 
requirements needs to be met to trigger the capture sequence. 

10 In one embodiment, the camera includes an interface such as an LCD screen 

and either buttons or a graphical interface so that the image profile thresholds can be 
entered into the camera. 

One method of generating an image profile suitable for initiating a capture 
sequence creates a mock up image that looks similar to the desired image. The 

15 mock up image can be created, for example, by using software to modify a 

background image. Alternatively, an image similar to that desired can be scanned 
into a computer, using well-known techniques. A histogram can then be generated 
from the mock up image using either software or hardware to model the histogram. 
The histogram of the mock up image can then be compared against the histogram of 

20 a background image captured by the image capture device. One or more thresholds 
based on the image metric(s) of the histograms can be set up so that a desired image 
will initiate a capture sequence. 
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For example, Figure 5 shows a representation of an example of an image that 
was created via a computer program. Figure 6 is a representative histogram of the 
luminosity corresponding to the image of Figure 5. In one embodiment, a 
background image is taken with the image capture device 201. The image is 
5 modified on host computer 270. A histogram of both the background image and the 
modified image are created using software residing on the host computer 270. The 
two histograms are compared against each other to determine which image metric(s) 
are best suited for use as a threshold to initiate a capture sequence. 

In the case of Figures 5 and 6, which correspond to lightning striking during 
10 the night, the count of the luminosity is clearly much higher at several levels than 
that of Figures 3 and 4. 

In one embodiment, a user manually determines one or more the image 
metric thresholds. In another embodiment, software determines the image metric 
threshold(s) by comparing the two or more histograms. The software determines 
1 5 one or more levels in the histograms in which the two image metrics differ the 
greatest. The software then creates one or more thresholds from the background 
image, making sure that the threshold is above the background image and allowing a 
margin to account for noise. The host computer 270 then communicates the 
threshold information to the image capture device 201 . The host computer can then 
20 be disconnected from the image capture device 201, if desired. 

Figure 10 shows an example of an image capture system set up to initiate a 
capture sequence based on the image metric of color content. This example initiates 
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the capture sequence based on blue content. However, the image profile can be set 
up based on other colorimetric or photometric data. 

The image information is obtained at block 1000. At block 1002 the image 
information is broken down into red, green and blue components. At block 1004, 
5 the blue component of the image data is processed to provide image metrics for the 
blue component. At block 1006, the blue component metrics are compared against 
the image profile. If the blue image metric(s) meet each of the one or more image 
profile thresholds (i.e., match the image profile) then the image is captured at block 
1010. If at block 1008, the blue image metric(s) do not meet all of the one or more 

10 thresholds, then the image is not captured. Instead, the flow sequence restarts at 
block 1000, and image information from another image is obtained. 

Thus, an apparatus and method determining when one or more images meet 
an image profile is disclosed. The specific arrangements and methods described 
herein are merely illustrative of the principles of this invention. Numerous 

15 modifications in form and detail may be made without departing from the scope of 
the described invention. Although this invention has been shown in relation to a 
particular embodiment, it should not be considered so limited. Rather, the described 
invention is limited only by the scope of the appended claims. 
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